Empirical Validation of Component-based Software Systems Generation and Evaluation Approaches

Component-based software development needs to formalize a process of generation, evaluation and selection of Composite COTS-based Software Systems (CCSS), enabling software architects to make early decisions; the Azimut approach and its associated software tool were proposed to tackle this problem. This article presents an experimental study conduced to compare Azimut approach with a Systematized Ad-Hoc approach, regarding generated solutions quality, cost and effort. Results suggest that: (1) Azimut generate better quality solutions at lower cost, but not statistically significant, and (2) there is strong evidence showing that the effort required is higher than for Systematized Ad-Hoc approach; re-sampling methods (Bootstrap and Jackknife) were applied to reinforce these conclusions. Also this study serves as a framework for validating approaches, process and tools for generating and evaluating component-based software systems.